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1  Objectives 

This  document  is  the  Final  Report  on  the  research  grant:  “Logical/Linear  Operators 
in  Early  Vision.”  The  overall  objectives,  as  stated  in  the  abstract  from  the  original 
proposal,  were  as  follows: 

We  propose  a  research  program  to  investigate  new  techniques  for  com¬ 
puting  image  descriptions  in  early  vision.  The  program  will  incorporate 
computational,  physiological,  and  psychophysical  aspects.  Research  will 
be  focused  on  the  certain  non-linearities  that  we  have  discovered  can  play 
an  important  role  in  improving  the  robustness  of  edge,  line,  and  curve  de¬ 
tection.  These  non-linearities  are  totally  diflFerent  from  those  previously 
studied  in  early  vision.  We  embody  them  in  what  we  call  logical/linear 
operators,  because  these  operators  exhibit  both  logical  (i.e.,  they  form 
an  appropriate  Boolean  algebra)  and  linear  properties.  We  submit  that 
these  operators  can  lead  to  both  more  robust  computer  vision,  and  to  new 
insights  into  biological  vision. 

The  research  program  has  matured  in  a  manner  that  exceeded  our  original  expecta¬ 
tions.  The  logical/linear  operator  abstraction  has  developed  a  particular  tangent-field 
model  that  has  developed  into  a  basis  for  texture  and  shading  analysis,  as  well  as  edge 
and  curve  detection,  it  has  provided  the  basis  for  a  new  approach  to  segmentation  and 
grouping,  and  it  has  provided  the  basis  for  a  new  approach  to  visual  shape  analysis. 
This  latter  study  has  created  enormous  interest  in  the  computer  vision  community, 
in  that  it  brought  the  mathematics  of  curve  evolution  to  shape  analysis  for  the  first 
time.  Finally,  and  perhaps  most  surprisingly,  mathematical  analysis  of  the  tangent 
abstraction  has  led  to  a  relationship  between  complexity  theory  and  segmentation, 
the  first  well-founded  approach  to  intermediate-level  problems  in  perceptual  group¬ 
ing.  Thus  we  have  the  theoretical  foundations  in  place  for  a  full  theory  of  vision, 
from  early  measurements  to  abstract  descriptions  of  generic  shape.  To  our  knowledge 
this  broad  covering  of  the  vision  problem  from  a  coherent  mathematical  perspective 
is  unique  in  the  field. 

The  research  program  has  matured  professionally  as  well.  The  logical/linear  op¬ 
erators  have  been  embedded  into  a  software  package  available  over  the  Wide  World 
Web,  and  hundreds  of  sites  around  the  world  have  ftp’d  a  copy  of  the  package.  This 
site  has  been  active  for  nearly  a  year,  which  is  a  short  time  for  such  material  to  diffuse 
into  the  research  community,  but  we  have  already  recieved  very  favorable  feedback 
regarding  it.  Furthermore,  Steven  Zucker  presented  a  public  lecture  series  at  the  Isaac 
Newton  Institute  for  Mathematical  Sciences  in  Cambridge  on  “Computational  Vision 
and  Biological  Perception” ,  which  covered  much  of  the  above  material.  These  lectures 
will  be  published  next  year  by  Princeton  University  Press  under  the  title:  “Textures 
and  Tangents” ,  and  it  will  be  the  second  volume  in  their  Newton  lecture  series.  (The 
first  volume  was  Hawking  and  Penrose’s  “The  Nature  of  Space  and  Time.”) 


2 


In  the  next  section,  we  provide  an  overview  of  this  tangent-map  approach  to 
vision,  and  a  few  example  figures.  Many  additional  examples  and  technical  details 
are  contained  in  the  accompanying  papers. 

2  Status  of  Effort 

It  is  difficult  to  express  our  satisfaction  with  the  past  three  years  of  research  without 
feeling  some  degree  of  embarassment.  The  logical/linear  operators  have  the  poten¬ 
tial  to  revitalize  early  vision,  the  complexity  analysis  approach  to  segmentation  has 
the  potential  to  provide  the  first  foundation  for  intermediate-level  vision  problems 
that  are  consistent  with  both  high-  and  low-level  processing,  and  the  curve-evolution 
approach  to  shape  analysis  has  played  a  key  catalytic  role  in  introducing  these  mathe¬ 
matical  techniques  to  the  community.  Although  curve-evolution  ideas  had  previously 
been  applied  to  image  processing  by  Osher,  Sethian,  and  Rudin  in  California,  their 
implications  for  vision  tasks  had  not  been  previously  realized.  Connecting  with  the 
non-linear  diffusion  community,  this  has  helped  to  create  a  movement  in  the  U.S., 
Canada,  and  Europe  to  explore  such  techniques.  Meetings  have  been  held  in  Cam¬ 
bridge,  UK,  Paris,  France,  Utrecht,  The  Netherlands,  Providence,  RI,  and  Palo  Alto, 
CA  of  this  community,  to  say  nothing  of  the  specialized  sessions  at  the  annual  con¬ 
ferences  in  our  field.  Thus  our  embarassment  is  positive  in  the  best  possible  sense. 


3  Accomplishments/New  Findings 

In  addition  to  completing  the  software  system  for  logical/linear  operators,  we  have 
continued  research  on  several  related  topics;  these  are  briefly  discussed  below. 

3.1  Logical/Linear  Operators 

Edge  detection  is  normally  considered  the  first  stage  of  image  analysis,  and  there  is 
little  doubt  that  one  of  the  major  frustrations  around  current  computer  vision  systems 
is  the  poor  quality  of  initial  edge  detection  and  the  subsequent  segmentations  they 
produce.  It  is  here  that  our  logical/linear  operators  apply  ([7]).  They  provide  more 
robust  local  estimates  of  edge  and  line  structure  by  enforcing  continuity  conditions 
along  the  tangential  extent  and  contrast  variations  along  the  normal  extent  of  their 
spatial  support.  (Recall  that  a  notion  of  orientation,  or  tangent,  is  essential  to  defining 
these  conditions.)  I  believe  these  operators  will  become  extremely  important  to  those 
portions  of  the  Air  Force’s  mission  that  involve  automatic  or  semi-automatic  analysis 
of  imagery,  as  well  as  to  related  civilian  technology  challenges. 

A  key  aspect  of  the  logical/linear  operators  is  that  they  formally  connect  edge 
structure  to  the  tangent  of  image  curves.  Thus  they  provide  an  abstract  description 
of  the  (differential)  geometric  structure  in  the  image  in  what  we  call  a  tangent  field. 
A  “portable”  version  of  a  system  to  support  these  applications  has  been  developed 
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by  Dr.  Lee  Iverson.  He  completed  a  first  implementation  as  a  Ph.  D.  student  here 
to  test  the  basic  ideas,  and  we  found  that  they  worked  extremely  well  on  complex 
biomedical  images  (e.g.,  angiograms  and  retinograms),  as  well  as  the  “standard  test 
images”  circulating  in  the  field  (e.g.,  “Paulina”);  see  Fig.  1. 


(a)  zoom  on 
Paolina’s  shoul¬ 
der 


(c)  Paolina’s 
shoulder  region 


(e)  Paolina:  origi¬ 
nal  image 
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(b)  zoom  on 
Paolina’s  hair 


(d)  Paolina’s  hair 
region 


map 


Figure  1:  An  illustration  of  the  tangent  map  created  by  logical/linear  operators  and  the 
subtlety  of  interpreting  (  “walking  through”  )it:  (top  row)  curves,  (bottom  row)  texture. 
The  original  image  in  (e)  and  the  full  edge  map  in  (f).  Notice  how  it  is  relatively  easy  to 
“walk”  from  one  tangent  to  another  when  the  tangents  depict  a  curve,  but  that  there  is 
substantial  confusion  for  the  more  complex  texture  (hair)  pattern.  For  many  of  the  tangents 
in  the  hair  it  is  functionally  impossible  to  determine  which  one  is  “next”  along  the  same 
hair.  Segmenting  curves  from  hair  is  based  on  a  complexity  analysis. 

This  figure  also  illustrates  the  intimate  connection  between  edge  detection  and 
segmentation  issues,  which  is  described  next. 


3.2  Complexity  Analysis  and  Segmentation 

We  illustrate  this  segmentation  problem  with  a  specific  example.  Imagine  standing 
on  an  edge  element  in  an  unknown  image,  as  in  the  first  Figure.  Is  this  edge  part  of 
a  curve,  or  perhaps  part  of  a  texture?  If  the  former,  which  is  the  next  element  along 
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the  curve?  If  the  pattern  is  a  texture,  is  it  a  hair  pattern,  in  which  nearby  elements 
are  oriented  similarly,  or  a  spaghetti  pattern,  in  which  they  are  not.  The  question  is 
in  part  one  of  complexity,  since  curves  are  “simpler”  than  textures  and  in  part  one  of 
dimensionality,  since  curves  are  1-D  and  textures  are  2-D. 

Benoit  Dubuc  and  I  (paper  appended)  have  proposed  a  measure  of  representational 
complexity  that  seeks  to  answer  these  questions,  and  it  involves  building  a  scale  space 
over  the  tangent  field.  We  sketch  these  ideas  in  this  Section. 

The  mathematical  foundation  for  our  measure  of  complexity  is  geometric  measure 
theory.  This  differs  from  classical  differential  geometry  in  that  curves  are  not  given 
as  prespecified  maps,  but  rather  we  use  a  parameterization-free  characterization  of 
curve-like  sets  as  those  with  Besicovitch  tangents;  informally,  these  are  the  locations 
in  the  image  at  which  contrast  organizes  into  locally  dense,  oriented  arrangements. 

Viewing  curves  as  curve-like  sets  of  points,  we  now  consider  dilations  over  it. 
Minkowski  dilations  are  routinely  used  in  mathematical  morphology  and  for  the  es¬ 
timation  of  fractal  dimension.  The  approach  consists  in  creating  a  new  set  which 
is  the  Minkowski  sum  with  a  dilating  (structuring)  element.  The  dilation  is  done 
isotropically  over  the  set.  More  formally,  given  a  set  £'  C  and  a  compact  convex 
set  F  C  R^,  the  dilation  oi  E  hy  F  is  given  by  E  0  F  =  {o  -I-  6,  a  e  E,b  G  F}, 
where  -f  here  denotes  the  vector  sum.  Given  e  >  0,  we  can  also  scale  the  structuring 
element  F  and  obtain  eF  =  {ex  :  x  e  F}.  An  example  of  the  isotropic  dilation  of  a 
line  segment  with  a  ball  is  shown  in  the  next  Figure. 

Our  complexity  measure  will  be  based  on  oriented  dilations  that  will  adapt  to  the 
local  structure  of  the  set.  Reconsidering  the  line  segment  example,  the  same  Figure 
illustrates  the  concepts  of  both  normal  and  tangential  dilations.  Notice  that  it  is 
the  tangent  interpretation  that  allows  oriented  dilations;  without  it,  such  an  analysis 
would  be  impossible. 


Isotropic  and  oriented  dilations,  (a)  isotropic  dilation  with  a  ball  of  radius  c.  (b)  normal  dilation 
(c)  tangential  dilation.  Oriented  dilations'  are  possible  because  of  the  intermediate  representation 
provided  by  the  Besicovitch  tangent  sets. 


The  departure  from  the  standard  Minkowski  dilation  approach  by  using  oriented 
dilations  will  be  essential  for  our  analysis  since  it  will  segregate  the  classification  curve 
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vs.  texture  (using  normal  dilations)  from  the  one  of  dust  (or  discontinuities)  vs.  curve 
(using  tangential  dilations).  In  particular,  we  shall  use  them  to  define  what  we  call 
the  normal  complexity  Cn{^)  and  the  tangential  complexity  Grid)  of  a  curve-like  set 
at  a  given  scale  5  >  0.  The  main  idea  is  to  look  at  the  rate  of  growth  of  the  measure 
of  the  dilated  sets.  In  the  case  of  normal  complexity,  it  will  be  the  area  of  Ei^{e).  If 
in  a  neighborhood  of  5  the  area  grows  like  S°‘,  then  we  say  that  the  normal  complexity 
Cn{S)  ioi  E  is  2 -a. 

Returning  to  the  Paolina  image,  we  observe  that,  in  the  hair  region,  the  rate  of 
growth  a  is  approximatively  0  at  the  chosen  scale  S,  thus  the  complexity  is  2  —  a  Ri  2 
(texture).  For  the  shoulder,  the  rate  of  growth  is  linear  (a  1),  leading  to  a  normal 
complexity  of  2  —  o;  w  1  (curve).  This  suggests  the  following  principles  for  segmenting 
the  tangent  field: 

dust:  sets  in  which  the  tangent  map  is  sparse,  the  object  almost  nowhere  extends 
along  its  length  locally,  and  might  be  thought  of  as  being  curve-free  (low  normal 
complexity  -  low  tangential  complexity); 

curves:  discrete  tangent  maps  for  which  a  curve  representation  is  completely  ade¬ 
quate.  Objects  extend  along  their  length,  like  Paolina’s  shoulder,  and  the  den¬ 
sity  of  other  tangents  is  low  almost  everywhere  along  it  in  a  local  neighborhood 
(low  normal  complexity  -  high  tangential  complexity); 

turbulence:  tangent  maps  that  are  characterized  by  objects  that  do  not  extend  along 
their  length  but  are  dense  in  the  normal  direction;  e.g.,  Paolina’s  uncombed  hair 
(high  normal  complexity  -  low  tangential  complexity); 

flow:  tangent  maps  for  which  the  objects  extend  along  their  length  and  are  also  dense 
in  the  normal  direction  (high  normal  complexity  -  high  tangential  complexity). 

The  result  of  this  segmentation  applied  to  the  Paolina  tangent  field  is  in  the  next 
Figure. 

The  analysis  thus  far  has  “focused”  (sic)  on  sharp  edges.  However,  vision  on 
natural  images  must  function  for  less  than  perfect  images  as  well,  a  topic  we  turn  to 
in  the  next  subsection. 

3.3  Blurry  Edges 

Not  all  edges  are  sharp,  and  there  is  significant  information  in  the  blur  profile  of  an 
edge.  James  Elder  and  I  have  generalized  the  logical/linear  framework  to  provide 
an  estimate  of  these  blur  profiles,  as  dictated  by  the  notion  of  “minimum  reliable 
scale”.  In  other  words,  with  edges  interpreted  as  tangents  to  image  curves,  they 
should  be  estimated  at  a  scale  which  is  sufficently  small  so  that  the  “straight-line” 
(or  tangent)  approximation  is  sensible  from  a  numerical  perspective  on  differential 
geometry,  but  at  a  scale  which  is  sufficently  large  that  the  estimates  are  reliabile  from 
a  statistical  perspective  on  signal  detection  theory.  This  extension  is  described  in 
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(c)  turbulence 


Figure  2:  Segmentation  can  be  pulled  back 
illustrates  the  four  structural  classes  of  image 
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1  .  I 


(b)  curves 


(d)  flow 


the  level  of  the  complexity  map.  This 


Figure  3;  Top  left:  The  blur  grade  is  linear,  ranging  from  =  1  pixel  to  at,  =  26.6 
pixels.  Top  right:  Edges  detected  by  local  scale  control  at  the  minimum  reliable 
scale.  Bottom  left:  Estimated  blur  scale.  Bottom  right:  Predicted  and  measured 
localization  precision. 


the  accompanying  papers.  Note  that  it  is  a  very  different  interpretation  of  the  role 
of  scale  than  is  common  in  computer  vision,  as  practiced  by  Witkin,  Koenderink, 
Lindeberg,  etc.  Its  reliability  against  an  edge  with  increasing  blur  is  shown  in  the 
next  Figure,  followed  by  an  illustration  of  how  it  can  be  used  to  segment  a  “sharp” 
tree  branch  from  a  field  of  defocussed  ones.  Clearly  such  problems  are  of  interest  to 
primates  swinging  through  a  forest,  and  to  robotic  vehicles  traversing  a  terrain  with 
imaging  sensors  of  finite  depth-of-field.  Note,  in  particular,  how  such  ideas  could  be 
used  for  quick  depth  estimation  in  such  vehicles.  The  full  paper  is  appended. 

A  surprising  consequence  of  the  minimum  reliable  scale  extension  is  that  it  opens 
an  entirely  new  class  of  algorithms  for  image  reconstruction  from  contour  codes.  An 
example  is  shown  in  the  next  figure;  details  are  in  the  paper  “Precision,  Blur,  and 
the  Perceptual  Content  of  the  Contour  Code.  We  mention  this  here  because  the 
question  of  image  communication  is  becoming  extremely  prominent  for  military  and 
civilian  applications,  and  several  companies  in  the  U.S.  have  begun  to  investigate 
this  new  approach;  see  Transitions.  The  key  for  us  was  understanding  the  deblurring 
inherent  in  traditional  reconstruction  algorithms:  note  how  shadows,  shading  and 
defocused  contours  appear  quite  unnatural.  It  appears  that  obtaining  perceptually 
accurate  reconstructions  from  the  contour  code  depend  upon  accurate  reblurring  of 
the  reconstructed  luminance  function.  The  extent  of  this  reblurring  is  dictated  by 
the  minimum  reliable  scale. 

Of  course,  some  blur  in  the  image  is  not  due  to  optical  effects,  but  is  due  to 
shading  effects.  Recalling  the  complexity  analysis  of  texture  flows,  we  now  introduce 
a  notion  of  shading  flow. 


Figure  4:  Top  left:  A  photograph  of  tree  branches  with  small  depth  of  field  (//3.5) 
and  near  focus.  Top  right:  Edge  map  computed  by  local  scale  control.  Middle 
left:  Foreground  structure  (focused  contours).  Middle  right:  Background  structure 
(defocused  contours).  Bottom  left:  Reconstructed  luminance  terrain  by  standard 
techniques  (Carlsson).  Bottom  right:  Reconstructed,  reblurred  result.  Note  how 
much  more  natural  this  reconstruction  appears. 


Figure  5:  Top  left:  Original  image.  Top  right:  Contours.  Bottom  left:  Recon¬ 
structed  luminance  function.  Bottom  right:  Reblurred  result. 
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3.4  Shading  Flow  Fields 

In  addition  to  the  applications-driven  activities,  we  are  pursuing  more  theoretical  do¬ 
mains  as  well.  The  logical/linear  “structural”  constraints  can  be  applied  at  a  larger 
scale  than  can  arise  from  contour  blurring;  for  example,  in  the  extended  shading 
distributions  that  are  common  in  natural  images.  The  analysis  of  such  surface  cov¬ 
erings  can  also  be  done  from  a  tangent-map-like  structure,  but  this  time  as  a  full 
two-dimensional  distribution.  Dr.  Pierre  Breton  has  completed  his  Ph.  D.  thesis  in 
this  area,  in  which  we  define  a  notion  of  “shading  flow  field”  as  the  basis  for  such  in¬ 
ferences.  It  derives  from  the  mathematical  observation  that  the  dual  flow  is  a  gradient 
fleld.  Image  gradients  are  of  course  important  to  shading  analysis,  and  provide  the 
basis  for  shape-from-shading  inferences.  An  illustration  follows,  indicating  both  how 
the  shading  flow  field  can  be  used  to  support  surface  inferences,  and  how  it  provides 
the  key  to  parsing  shadows  and  other  light-source  changes  from  images.  Details  are 
in  the  attached  paper:  “Shadows  and  Shading  Flow  Fields” . 


Figure  6:  Top-left:  an  image  for  which  assumptions  of  standard  shape-from-shading  (Bischel  and 
Pentland,  1992)  are  violated.  Top-right:  neither  of  the  two  most  plausible  scenes  (the  surface  albedo 
changes  abruptly  or  a  semi-transparent  object  lies  somewhere  in  the  scene)  can  be  implied  from  the 
depth  map  obtained.  Bottom-left:  the  shading  flow  field  is  depicted  by  thin  arrows  and  the  tangent 
field,  by  thick  arrows.  Bottom-right:  the  accurate  shape  recovery  illustrated  by  the  surface  normals 
is  obtained  from  the  shading  flow  field. 

Dr.  M.  Langer  has  completed  a  Ph.  D.  thesis  on  a  very  different  aspect  of  how 
light  interaction  with  surfaces.  The  classical  model  is  derived  by  modeling  how  image 
contrast  changes  as  a  function  of  surface  orientation.  Our  new  perspective  is  obtained 
by  modeling  how  light  propagates  through  a  scene;  i.e.,  by  modeling  light  rays  instead 
of  surface  patches.  This  has  given  rise  to  an  entirely  new  class  of  shape  from  shading 
algorithms,  which  are  naturally  parallel  (for  SIMD  computers).  Quite  surprisingly, 
these  methods  are  analagous  in  a  certain  technical  sense  to  what  is  required  for 
modeling  (free)space  for  robot  navigation  and  to  do  computer  graphics  on  parallel 
(SIMD)  machines. 
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3.5  Contour  Completion 

An  outline  drawing  often  serves  as  an  excellent  depiction  of  a  visual  scene,  and  some¬ 
how  our  visual  systems  can  form  two-  and  three-dimensional  percepts  solely  from 
one-dimensional  contour  information.  The  complexity  analysis  showed  how,  for  the 
first  time,  those  tangents  belonging  to  image  contours  could  be  segmented  from  those 
belonging  to  textures  and  other  images  structures.  Thus  we  are  now  in  a  position  to 
consider  how  to  link  these  together. 

In  mathematics,  contour  closure  plays  a  key  role  in  bridging  this  gap,  however 
in  perception  the  link  between  closure  and  shape  is  unclear.  James  Elder  and  I 
have  been  attempting  to  better  understand  this  relationship.  We  devised  a  set  of 
visual  search  experiments  in  which  subjects  discriminate  outline  figures  by  means  of 
their  two-dimensional  shape.  By  modulating  the  degree  of  closure  of  the  outlines,  we 
were  able  to  show  that  two-dimensional  shape  processing  is  rapid  for  closed  stimuli 
but  slow  for  open  stimuli.  We  further  show  that  search  speed  can  be  characterized 
as  a  smooth,  monotonic  function  of  the  degree  of  closure,  supporting  the  notion  of 
a  perceptual  closure  continuum.  We  have  just  begun  to  translate  these  ideas  into 
computer  algorithms,  the  first  paper  for  which  is  appended  (“Computing  Contour 
Closure”). 

From  such  algorithms  we  should  be  able  to  find  the  bounding  contours  of  objects 
and  major  components  within  those  objects,  which  leads  us  to  the  final  topic,  visual 
shape  analysis. 

3.6  Descriptions  of  Visual  Shape 

The  analysis  of  visual  shape  naturally  decomposes  into  a  two-stage  process,  the  first 
of  which  provides  a  general  shape  description  to  serve  as  a  base  key  into  categorical 
possibilities,  and  the  second  of  which  identifies  specific  members  of  the  category. 
For  example,  an  organization  of  a  small  blob  on  a  large  blob  with  several  elongated 
structures  emanating  from  the  large  blob  might  be  categorized  as  a  person,  and  then 
that  person  might  be  identified  as  “Richard  Nixon” .  We  shall  concentrate  on  the  first, 
categorical  stage,  since  this  is  the  stage  that  has  been  most  frustrating  for  researchers 
in  computer  vision.  In  fact,  we  believe  this  to  be  the  first  mathematically-founded 
approach  to  generic  shape  analysis. 

The  analysis  builds  directly  on  that  of  the  previous  sections.  Assume  a  tangent 
field  has  been  inferred,  and  those  tangents  comprising  a  1-dimensional  curve  have 
been  selected.  It  is  then  necessary  to  infer  a  curve  through  this  tangent  field,  and  the 
technique  in  the  Elder  and  Zucker  paper,  or  in  [3]  sulRce. 

How  can  the  generic  2-dimensional  shape  bounded  by  this  closed  curve  be  de¬ 
scribed?  What  are  the  natural  parts,  and  are  there  other  components  to  such  a  de¬ 
scription?  Biederman’s  “geons”  ([1]),  Hoffman  and  Richards’  “codons”  ([6]),  Blum’s 
medial  axis  transform  ([2]),  and  Marr  and  Nishihara’s  “stick  figures”  ([11])  are  pos¬ 
sibilities,  but  they  all  differ.  Some  concentrate  on  boundary  information,  and  others 
on  interior  (region)  information.  Each  has  a  regime  of  figures  on  which  they  work 
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Figure  7:  The  points  on  the  initial  ciu've  A  move  to  B  to  generate  a  new  curve.  The 
direction  and  magnitude  of  this  motion  is  arbitrary  in  order  to  capture  general  deformations. 
However,  with  mild  restrictions  appropriate  to  shape,  one  can  classify  this  deformation  to 
be  a  sum  of  constant  deformation  and  curvature  deformation  along  the  normal. 

well,  e.g.,  machined  industrial  components  for  geons  and  leaves  for  Blum,  but  none 
provides  a  unifying  methodology.  Moreover,  none  provide  a  natural  scale-space  for 
significance  of  the  components,  in  the  sense  that  one  can  derive  why  the  head  is  less 
significant  than  the  torso,  etc.  This  significance  ordering  requires  a  compositional 
component,  as  was  earlier  attempted  in  picture  grammers  ([13];  [10]). 

Our  approach  to  categorical  shape  description  derives  from  the  theory  of  curve 
evolution.  This  is  a  well-studied  area  of  mathematics,  for  which  a  beautiful  theory 
has  been  developed  over  the  past  half  century  ([5],  [14,  12],  [9],  [4].  What  will  be  of 
primary  significance  for  us  is  that  such  evolutions  develop  singularities — or  shocks — 
and  these  singularities  define  the  natural  shape  descriptors.  They  also  imply  a  natural 
significance  hierarchy  over  shapes,  which  is  the  structural  scale  space  that  we  seek.  It 
is  this  specific  connection  between  the  singularities  of  the  curve  evolution  process  and 
the  visual  analysis  of  shape  that  is  turning  out  to  be  so  seminal  for  these  applications 
(cf.  the  opening  comments  in  the  Sec.  “Status  of  Effort”.) 

We  begin  with  the  priciple  that:  slight  changes  in  the  boundary  of  an  object  cause 
only  slight  changes  to  its  shape,  which  suggests  that  we  consider  slight  boundary  de¬ 
formations.  Let  the  shape  be  represented  by  the  curve  Co{s)  =  {xo{s),yo{s)),  where  s 
is  the  parameter  along  the  curve  (not  necessarily  arclength),  Xq  and  yo  are  the  Cartex- 
ian  coordinats  and  the  subscript  )  denotes  the  initial  curve  prior  to  deformation.  We 
consider  the  specific  deformation  (see  Kimia  et  al.  ([8])  for  derivation,  background, 
and  related  references): 


dt 

C{s,0) 


(1  -h  eK)N 
Co{s) 


12 


An  illustration  of  a  curve  evolving  according  to  this  equation  is  shown  in  the  next 
Figure.  The  e  parameter  runs  along  the  x-axis,  and  the  time  of  evolution,  or  scale, 
along  the  vertical  axis.  The  result  is  a  scale  space  built  upon  the  shape,  which  we 
call  a  reaction/ diffusion  space. 

This  space  is  no  named  because  of  the  parameter  e,  which  controls  the  amount  of 
diffusion  during  the  evolution.  For  large  amounts  of  e  it  dominates,  and  the  result  is 
a  geometric  heat  equation. 

At  the  opposite  extreme,  when  e  =  0,  there  is  no  smoothing  and  shocks  are 
created.  Four  such  shocks  are  illustrated  in  the  Figure;  they  are; 

1.  First-Order  Shocks  are  discontinuities  in  orientation  of  the  boundary  of  a  shape. 
They  denote  protrusions  and  intrusions,  such  a  the  nose  on  a  face. 

2.  Second-Order  Shocks  result  when,  during  the  process  of  deformation,  two  dis¬ 
tinct  non-neighboring  boundary  points  join  and  not  all  the  other  neighboring 
boundary  points  have  collapsed  together.  Second-order  shocks  define  the  parts 
of  a  shape. 

3.  Third-Order  Shocks  result  when,  during  the  process  of  deformation,  two  dis¬ 
tinct  non-neighboring  boundary  points  join  in  a  manner  such  that  neighboring 
boundaries  of  each  point  also  collapse  together.  Third-order  shocks  indicate  an 
extended  axis,  as  in  a  tail  or  a  bend. 

4.  Fourth- Order  Shocks  occur  when,  during  the  process  of  deformation,  a  closed 
boundary  collapses  to  a  point.  Thus  fourth-order  shocks  are  the  seeds  of  a 
shape,  in  that  they  indicate  the  locations  where  mass  is  to  be  attached. 

Second-order  shocks  persist  for  only  an  instant  in  time,  and  then  give  rise  to  a 
pair  of  first-order  shocks  traversing  in  opposite  directions.  The  locus  of  points  traced 
out  by  the  shocks  is  the  Blum  skeleton. 

Because  the  shocks  occur  during  an  evolutionary  process,  the  time  of  formation  is 
related  to  significance.  In  particular,  the  least  significant  shocks  form  first,  and  the 
most  significant  ones  last.  The  result  is  a  hierarchical  description  that  is  suitable  for 
categorical  object  description;  see  Figure. 

The  most  recent  activity  has  related  this  class  of  shape  description  to  the  fragment 
grouping  problem  via  a  principle  of  perceptual  occlusion.  Again,  the  connection  is 
formal,  and  is  described  in  the  accompanying  paper  by  August  and  Zucker. 

3.7  Summary  of  Contributions 

In  this  Section  we  have  attempted  to  outline  the  contributions  made  during  the  tenure 
of  this  grant.  They  range  from  initial  image  measurement  via  logical/linear  operators, 
and  their  extension  to  blurry  edges,  to  an  intermeiate-level  theory  of  complexity- 
based  segmentation  to,  finally,  a  high-level  theory  of  shape  description.  A  notion  of 
orientation,  or  tangent,  is  central  to  each  of  these  levels. 
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Figure  8:  An  illustration  of  a  portion  of  the  reaction-diffusion  space  for  a  DOLL  image 
(from  the  National  Research  Council  of  Canada’s  Laser  Range  Image  Library  CNRC9077 
Cat  No  422.  The  image  was  thresholded  and  stored  as  a  128x128  image.  The  numbers  on 
the  a;-axis  are  indicative  of  the  ratio  of  reaction  to  diffusion.  Note  how  the  reaction  creates 
shocks,  and  the  diffusion  “melts”  the  boundary  of  the  shape.  After  [8]. 
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First-Order  Shock 


Second-Order  Shock 


Figure  9:  Illustration  of  the  four  types  of  shocks  formed  in  the  course  of  the  reactive 
deformation  process;  each  is  correlated  with  a  generic  shape  category,  i.e.,  protrusion,  part, 
bend  and  seed.  For  each  shock  type,  the  maximal  inscribed  disc  is  overlayed.  After  [8]. 
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Figure  10:  a)  The  evolution  of  shocks  leads  to  parts,  protrusions,  and  bends.  Again  the 
DOLL  image  is  shown.  The  contour  in  box  N  corresponds  to  increasing  boundary  evolution 
(time)  steps.  Observe  that  the  “feet”  partition  from  the  “legs”  (via  second-order  shocks) 
between  frames  3  and  4,  and  the  “hands”  from  the  “arms”  between  frames  2  and  3.  Following 
these  second-order  shocks,  first-order  shocks  develop  as  the  “arms”  are  “absorbed”  into  the 
chest.  Running  this  process  in  the  other  direction  would  illustrate  how  the  arms  “protrude” 
from  the  chest.  b)  The  hierarchical  decomposition  of  a  doll  into  parts.  Selected 

frames  were  organized  into  a  hierarchy  according  to  the  principle  that  the  significance  of  a 
part  is  directly  proportional  to  its  survival  duration.  After  [8] 


15 


4  Personnel  Supported 

The  following  graduate  students  and  Post-Doctoral  Fellows  have  been  supported  in 
whole  or  in  part  by  this  grant: 

•  Pierre  Breton 

•  Benoit  Dubuc 

•  Douglas  Miller 

•  Michael  L  anger 

•  Jonas  August 

•  James  Elder 

•  Michael  Langer 

In  addition,  Dr.  Lee  Iverson  has  remained  in  a  consultative  capacity. 


5  Publications 

The  following  papers  were  published  during  the  previous  36  months,  or  the  tenure  of 
the  grant. 

1.  Kimia,  B.,  and  Zucker,  S.W.,  Symbolic  inverse  to  Gaussian  Blur,  Optical  Engi¬ 
neering,  1993,  32(1),  166  -  176. 

2.  Elder,  J.,  and  Zucker,  S.W.,  Contour  closure  and  the  perception  of  shape.  Vision 
Research,  1993,  33(7),  981  -  991. 

3.  Zucker,  S.W.,  Endowing  AI  with  Vision:  A  biological  and  computational  per¬ 
spective,  Current  Science,  1993,  64(6),  407  -  418. 

4.  79.  Kimia,  B.,  Tannenbaum,  A.,  and  Zucker,  S.W.,  Shapes,  shocks,  and  defor¬ 
mations,  I,  Int.  J.  Computer  Vision,  15,  1995,  189  -  224. 

5.  101.  Langer,  M.,  and  Zucker,  S.W.,  Diffuse  Shading,  Visibility  Fields,  and  the 
Geometry  of  Ambient  Light,  Proc.  Fourth  Int.  Conf.  on  Computer  Vision, 
Berlin,  1993,  138  -  147. 

6.  102.  Zucker,  S.W.,  Jones,  D.,  and  Yeshurun,  H.,  Random  dot  Moire  stereograms 
indicate  a  distributed  code  for  orientation  and  disparity.  Annual  Meeting,  The 
Association  for  Research  in  Vision  and  Opthalamology  (ARVO),  Florida,  May, 
1993;  Investigative  Opth.  and  Vis.  Science,  1439. 


16 


7.  103.  Miller,  D.,  and  Zucker,  S.W.,  Analog  models  of  visual  cortex  and  the 
number  of  storable  states:  A  basis  for  hypteracuity,  Snowbird  Workshop  on 
Neural  Computing,  April,  1993,  17. 

8.  104.  Kimia,  B.,  Tannenbaum,  A.,  and  Zucker,  S.W.,  Non-linear  shape  approx¬ 
imation  via  the  entropy  scale  space,  Proc.  SPIE  Conf.  on  Geometric  Methods 
in  Computer  Vision,  II,  San  Diego,  July,  1993,  vol.  2031,  218  -  233. 

9.  105.  Jones,  D.,  and  Zucker,  S.W.,  A  distributed  code  for  disparity  and  orienta¬ 
tion,  Proc.  NATO  Advanced  Research  Workshop  on  Binocular  Stereopsis  and 
Optic  Flow,  York  University,  Toronto,  June,  1993. 

10.  Langer,  M.,  and  Zucker,  S.W.,  Shape  from  Shading  on  a  cloudy  day,  J.  Opt. 
Soc.  Am.,  A,  1994,  11(2),  467  -  478. 

11.  77.  Dubuc,  B.,  Zucker,  S-W.,  and  Stryker,  M.,  Complexity  Maps  Reveal  Clus¬ 
ters  in  Neuronal  Arborizations,  Fractals,  2(6),  1994,  2(2),  297  -  301. 

12.  78.  Elder,  J.,  and  Zucker,  S.W.,  A  measure  of  closure.  Vision  Research,  34, 
1994,  3361  -  3370. 

13.  106.  Langer,  M.S.,  and  Zucker,  S.W.,  Spatially  varying  illumination:  A  compu¬ 
tational  model  of  converging  and  diverging  sources,  Proc.  Third  European  Conf. 
on  Computer  Vision,  Stockholm,  1994;  in  J.-O.  Eklundh  (ed.).  Computer  Vision 

-  ECCV  ’94,  Springer- Verlag,  New  York,  1994,  227  -  232. 

14.  107.  Dubuc,  B.,  Zucker,  S.W.,  and  Stryker,  M.,  Complexity  Maps  Reveal  Clus¬ 
ters  in  Neuronal  Arborizations,  Fractals  in  Engineering  ’94,  Montreal,  June  1-4, 
1994, 

15.  108.  Dubuc,  B.,  and  Zucker,  S.W.,  Curve-like  Sets,  Normal  Complexity,  and 
Representation,  International  Conf.  on  Pattern  Recognition,  Jerusalem,  1994, 
A  216-221. 

16.  Kimia,  B.,  Tannenbaum,  A.,  and  Zucker,  S.W.,  On  the  shape  triangle,  in  C. 
Arcelli,  L.  Cordelia,  and  G.  Sanniti  di  Baja  (eds).  Aspects  of  Visual  Form 
Processing,  1994,  World  Scientific,  Singapore,  307  -  323. 

17.  Iverson,  L.,  and  Zucker,  S.W.,  Logical/linear  operators  for  image  curves,  IEEE 
Trans.  Pattern  Analysis  and  Machine  Intelligence,  17(10),  1995,  982  -  996. 

18.  Breton,  P.,  and  Zucker,  S.W.,  Contours,  Shading  Flows,  and  Generalized  Shape 
from  Shading,  Proc.  Vision  Interface  95,  Quebec  City,  Canada,  May,  1995,  71 

-  80. 

19.  Dubuc,  B.,  and  Zucker,  S.W.,  La  Carte  Complexity  et  son  Impact  sur  le  Choix 
de  Representation,  Proc.  Vision  Interface  95,  Quebec  City,  Canada,  May,  1995, 
186  -  194.  Awarded  Best  Paper  Prize. 


17 


20.  Langer,  M.S.,  Breton,  R,  and  Zucker,  S.W.,  Massively  Parallel  Radiosity  in  the 
Presence  of  Multiple  Isotropic  Volume  Scattering,  Proc.  Graphics  Interface  95, 
Quebec  City,  Canada,  May,  1995,  103  -  108. 

21.  Elder,  J.,  and  Zucker,  S.W.  Scale-space  surfaces  and  blur  estimation,  Proc. 
Vision  Interface  95,  Quebec  City,  Canada,  May,  1995,  140  -  147.  Awarded  Best 
Paper  Prize. 

22.  Dubuc,  B.,  and  Zucker,  S.W.,  Indexing  visual  representations  through  the  com¬ 
plexity  map,  Proc.  Fifth  International  Conf.  on  Computer  Vision,  Cambridge, 
MA,  June,  1995. 

23.  Langer,  M.,  and  Zucker,  S.W.,  A  ray-based  computational  model  of  light  sources 
and  illumination,  IEEE  Workshop  on  Physics-Based  Modeling,  MIT,  Cam¬ 
bridge,  MA,  June,  1995. 

24.  Elder,  J.,  and  Zucker,  The  local  character  of  generalized  luminance  transitions. 
Annual  Meeting,  Association  for  Research  in  Vision  and  Opthalmology  (ARVO), 
Fort  Lauderdale,  FL.,  May,  1995.;  Investigative  Opth.  and  Vis.  Science,  36(4), 
1995,  p.  S836. 

25.  Langer,  M.,  Dudek,  G.,  and  Zucker,  S.W.,  Space  occupancy  using  multiple 
shadow  images,  IROS  95,  lEEE/RSJ  Int.  Conf.  on  Intelligent  Robots  and 
Systems,  Pittsburgh,  PA,  August,  1995. 

26.  Elder,  J.,  and  Zucker,  S.W.  Shadows,  defocus,  and  reliable  estimation,  CAIP 
’95,  Prague,  September,  1995;  in  V.  Hlavac  and  R.  Sara  (eds.).  Computer 
Analysis  of  Images  and  Patterns,  Lecture  Notes  in  Computer  Science  970, 
Springer  Verlag,  318  -  325,  1995. 

27.  Dubuc,  B.,  and  Zucker,  S.W.,  A  complexity  space  for  curve  grouping,  CAIP 
’95,  Prague,  September,  1995;  in  V.  Hlavac  and  R.  Sara  (eds.).  Computer 
Analysis  of  Images  and  Patterns,  Lecture  Notes  in  Computer  Science  970, 
Springer  Verlag,  33  -  40,  1995. 

28.  Elder,  J.,  and  Zucker,  S.W.,  Local  scale  control  for  edge  detection  and  blur 
estimation,  Proc.  Fourth  European  Conf.  on  Computer  Vision,  Cambridge, 
U.K.,  1996;  in  Buxton,  B.,  and  Cipolla,  R.  (eds).  Computer  Vision  -  ECCV 
’96,  Springer,  1996,  vol.  2,  57  -  69. 

29.  Elder,  J.,  and  Zucker,  S.W.,  Computing  Contour  Closure,  Proc.  Fourth  Euro¬ 
pean  Conf  on  Computer  Vision,  Cambridge,  U.K.,  1996;  in  Buxton,  B.,  and 
Cipolla,  R.  (eds).  Computer  Vision  -  ECCV  ’96,  Springer,  1996,  vol.  1,  399 
-  412. 


30.  Elder,  J.,  and  Zucker,  S.W.,  The  computation  of  closed  bounding  contours, 
Proc.  Vision  Interface  96,  Toronto,  May  21-24,  1996. 


18 


31.  Elder,  J.,  and  Zucker,  S.W.,  Precision,  Blur,  and  the  perceptual  content  of  a 
contour  code,  Proc.  IEEE  Conf.  on  Computer  Vision  and  Pattern  Recognition 
-  CVPR  ’96,  San  Francisco,  CA,  June,  1996. 

32.  Allman,  J.,  and  Zucker,  S.W.,  On  cytochrome  oxidase  blobs  in  visual  cortex,,  in 
Harris,  L.,  and  Jenkin,  M.  (eds.).  Spatial  Vision  for  Humans  and  Robots, 
Cambridge  University  Press,  New  York,  1993,  33  -  39. 

33.  Zucker,  S.W.,  Iverson,  L.,  and  Dobbins,  A.,  From  curve  detection  to  shape  de¬ 
scription,  in  A.  Basu  and  X.  Li  (eds.).  Computer  Vision:  systems,  theory, 
and  applications.  World  Scientific,  Singapore,  1993,  25  -  40. 

34.  Zucker,  S.W.,  Visual  Computations  and  Dot  Clusters,  in  I.  Cox  and  B.  Julesz 
(eds.).  Partitioning  Data  Sets,  American  Mathematical  Society,  Providence, 

1995,  391  -  408. 

35.  Kimia,  B.,  Tannenbaum,  A.,  and  Zucker,  S.W.,  On  Optimal  Control  Meth¬ 
ods  in  Computer  Vision  and  Image  Processing,  in  B.  ter  Haar  Romeny.(ed.) 
Geometry-Driven  Diffusion  in  Computer  Vision,  Kluwer,  1994,  307  - 
338. 

36.  Kimia,  B.,  Tannenbaum,  A.,  and  Zucker,  S.W.,  Exploring  the  shape  manifold: 
the  role  of  conservation  laws,  in  Ying-Lie,  O.,  Toet,  A.,  Foster,  D.,  Heijmans, 
H.,  and  Meer,  P.  (eds).  Shape  in  Picture,  Springer- verlag,  1994,  601  -  620. 

37.  Zucker,  S.W.,  Perceptual  Grouping,  in  Arbib,  M.  (ed.)  Handbook  of  Brain 
Theory  and  Neural  Networks,  MIT  Press,  Cambridge,  1995,  725  -  727. 

38.  Zucker,  S.W.,  Structural  scales  in  computational  vision,  in  K.  Boyer  and  N. 
Ahuja  (eds).  Advances  in  Computer  Vision,  1996,  IEEE  Computer  Society 
Press. 

39.  Langer,  M.,  and  Zucker,  S.W.,  A  Ray-based,  computational  model  of  spatially 
varying  illumination.  Computer  Vision  and  Image  Understanding,  accepted, 

1996. 

6  Interactions  /  Transitions 

6.1  Participations/presentations 

The  following  Invited  Lectures  and  Presentations  were  given  during  the  duration  of 
the  current  Grant. 

1.  Distinguished  AI  Colloquium,  Ohio  State  University,  Columbus,  May,  1996. 


19 


2.  Opening  Lecturer  (invited),  Workshop  on  Mathematical  Methods  in  Computer 
Vision,  The  Geometry  Center,  University  of  Minnesota,  Minneapolis,  Sept., 
1995. 

3.  Invited  Lecturer,  Workshop  on  Computational  Neuroscience,  Marine  Biological 
Laboratories,  Woods  Hole,  Ma,  August,  1995. 

4.  Invited  Lecturer,  Methods  in  Computational  Neuroscience,'  Marine  Biological 
Laboratories,  Woods  Hole,  August,  Ma,  1995. 

5.  Invited  Lecturer,  Stockholm  Workshop  on  Computer  Vision,  Rosenon,  Sweden, 
August,  1995. 

6.  Invited  Lecturer,  NECI  Vision  Workshop,  Princeton,  N.J.,  March,  1995. 

7.  Invited  Lecturer,  30-th  Anniversary,  Computer  Vision  Laboratory,  University 
of  Maryland,  College  Park,  1994. 

8.  Invited  Lecturer,  Second  Workshop  on  Visual  Form,  International  Association 
for  Pattern  Recognition,  Capri,  May,  1994. 

9.  Invited  Lecturer,  Workshop  on  Computational  Neuroscience,  Marine  Biological 
Laboratories,  Woods  Hole,  Ma,  1994. 

10.  Invited  Lecturer,  The  Hebb  Symposium  on  Neurons  and  Biological  Dynamics, 
The  Fields  Institute  for  Mathematical  Sciences,  University  of  Toronto,  May, 
1994. 

11.  Invited  Lecturer,  U.S.  -  Israel  Binational  Workshop  on  Cerebral  Cortex  and 
Object  Perception,  Jerusalem,  March,  1994. 

12.  Newton  Institute  Public  Lecture  Series,  Computational  Vision  and  Biological 
Perception,  University  of  Cambridge,  October  -  Novermber,  1993. 

13.  Invited  Lecturer,  Fourth  Annual  British  Machine  Vision  Association  Meeting  on 
Natural  and  Machine  Vision,  British  Institute  of  Radiology,  London,  October 
1993. 

14.  Invited  Lecturer,  Workshop  on  Computational  Neuroscience,  Marine  Biological 
Laboratories,  Woods  Hole,  Ma,  August,  1993. 

15.  Invited  Lecturer,  Workshop  on  Mechanisms  of  Visual  Object  Recognition,  Santa 
Fe  Institute,  Santa  Fe,  NM,  August,  1993. 
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6.2  Transitions 

The  logical/linear  operator  system  for  local  edge,  bright  line,  and  dark  line  detection 
has  reached  a  sufficently  stable  status  that  the  software  has  been  made  available  for 
the  community.  Access  is  free,  and  is  distributed  via  anonymous  ftp  over  the  internet 
from  a  server  at  McGill’s  Center  for  Intelligent  Machines.  The  distributed  software 
runs  on  Sun  Microsystems  and  Silicon  Graphics  platforms.  Previous  versions  were 
tested  here,  at  SRI  International,  and  at  Brown  University.  Dr.  Iverson  is  now  a 
Research  Scientist  at  SRI  International,  in  Palo  Alto;  working  as  a  consultant  on  this 
grant,  he  has  re-implemented  the  entire  system  to  be  more  efficent  and  portable. 

The  response  to  this  system  has  been  fantastic.  A  giant  log  file  of  the  different 
sites  that  have  accessed  the  system  is  available,  should  it  be  desired. 

7  New  Discoveries,  Inventions,  Patent  Disclosures 


None. 


8  Honors  and  Awards 

•  Nominated,  Best  Paper  Prize  (with  J.  August,  K.  Siddiqi)  for  the  paper:  Frag¬ 
ment  grouping  via  the  principle  of  perceptual  occlusion.  International  Confer¬ 
ence  on  Pattern  Recognition,  Vienna,  August,  1996. 

•  Best  Paper  Prize  (with  J.  Elder)  for  the  paper:  Scale-space  surfaces  and  blur 
estimation,  at  Vision  Interface  95,  Quebec  City,  Canada,  May,  1995. 

•  Best  Paper  Prize  (with  B.  Dubuc)  for  the  paper:  La  carte  complexite  et  son 
impact  sur  le  choix  de  representation,  Vision  Interface  95,  Quebec  City,  Canada, 
May,  1995. 

•  Visiting  Professor  (invited),  Sloan  Center  for  Theoretical  Neurobiology,  Univer¬ 
sity  of  California  at  San  Francisco,  1995  -  1998. 

•  D.  W.  Ambridge  Award,  1993,  for  Lee  Iverson’s  Ph.  D.  thesis. 

•  Elected  (By)Fellow,  Churchill  College,  Cambridge,  1993. 

•  SERC  Fellow,  Newton  Institute  for  Mathematical  Sciences,  University  of  Cam¬ 
bridge,  1993. 

•  Invited  Short-Term  Visitor,  Institute  for  Research  in  Cognitive  Science,  Univer¬ 
sity  of  Pennsylvania,  Philadelphia,  March,  -  April,  1993. 

•  Invited  Visiting  Professor,  School  of  Mathematical  Sciences,  Tel  Aviv  Univer¬ 
sity,  December,  1992  -  January,  1993. 
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•  Fellow,  Canadian  Institute  for  Advanced  Research. 

•  Fellow,  Institute  of  Electrical  and  Electronic  Engineers  (IEEE) . 
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